近几十年来,随着数据的可用,统计语言学已大大提高。这使研究人员能够研究语言的统计特性如何随时间变化。在这项工作中,我们使用来自Twitter的数据来探索英语和西班牙语,考虑到不同尺度的排名多样性:时间(从3到96小时),空间(从3公里到3000+km Radii)和语法(从字母组到五角形到Pentagrams) )。我们发现所有三个量表都是相关的。但是,最大的变化来自语法量表的变化。在最低的语法量表(会标)上,排名多样性曲线最相似,独立于其他量表,语言和国家的价值。随着语法量表的增长,等级多样性曲线的变化更大,具体取决于时间和空间量表以及语言和国家。我们还研究了Twitter特定令牌的统计数据:表情符号,主题标签和用户提及。这些特殊类型的令牌表现出一种sigmoid的行为作为等级多样性函数。我们的结果有助于量化似乎普遍存在的语言统计数据的各个方面,这可能导致变化。
translated by 谷歌翻译
Timely and effective response to humanitarian crises requires quick and accurate analysis of large amounts of text data - a process that can highly benefit from expert-assisted NLP systems trained on validated and annotated data in the humanitarian response domain. To enable creation of such NLP systems, we introduce and release HumSet, a novel and rich multilingual dataset of humanitarian response documents annotated by experts in the humanitarian response community. The dataset provides documents in three languages (English, French, Spanish) and covers a variety of humanitarian crises from 2018 to 2021 across the globe. For each document, HUMSET provides selected snippets (entries) as well as assigned classes to each entry annotated using common humanitarian information analysis frameworks. HUMSET also provides novel and challenging entry extraction and multi-label entry classification tasks. In this paper, we take a first step towards approaching these tasks and conduct a set of experiments on Pre-trained Language Models (PLM) to establish strong baselines for future research in this domain. The dataset is available at https://blog.thedeep.io/humset/.
translated by 谷歌翻译